25 research outputs found
Scampi: a robust approximate message-passing framework for compressive imaging
Reconstruction of images from noisy linear measurements is a core problem in
image processing, for which convex optimization methods based on total
variation (TV) minimization have been the long-standing state-of-the-art. We
present an alternative probabilistic reconstruction procedure based on
approximate message-passing, Scampi, which operates in the compressive regime,
where the inverse imaging problem is underdetermined. While the proposed method
is related to the recently proposed GrAMPA algorithm of Borgerding, Schniter,
and Rangan, we further develop the probabilistic approach to compressive
imaging by introducing an expectation-maximizaiton learning of model
parameters, making the Scampi robust to model uncertainties. Additionally, our
numerical experiments indicate that Scampi can provide reconstruction
performance superior to both GrAMPA as well as convex approaches to TV
reconstruction. Finally, through exhaustive best-case experiments, we show that
in many cases the maximal performance of both Scampi and convex TV can be quite
close, even though the approaches are a prori distinct. The theoretical reasons
for this correspondence remain an open question. Nevertheless, the proposed
algorithm remains more practical, as it requires far less parameter tuning to
perform optimally.Comment: Presented at the 2015 International Meeting on High-Dimensional Data
Driven Science, Kyoto, Japa
Approximate Message Passing with Restricted Boltzmann Machine Priors
Approximate Message Passing (AMP) has been shown to be an excellent
statistical approach to signal inference and compressed sensing problem. The
AMP framework provides modularity in the choice of signal prior; here we
propose a hierarchical form of the Gauss-Bernouilli prior which utilizes a
Restricted Boltzmann Machine (RBM) trained on the signal support to push
reconstruction performance beyond that of simple iid priors for signals whose
support can be well represented by a trained binary RBM. We present and analyze
two methods of RBM factorization and demonstrate how these affect signal
reconstruction performance within our proposed algorithm. Finally, using the
MNIST handwritten digit dataset, we show experimentally that using an RBM
allows AMP to approach oracle-support performance
Sparse Estimation with the Swept Approximated Message-Passing Algorithm
Approximate Message Passing (AMP) has been shown to be a superior method for
inference problems, such as the recovery of signals from sets of noisy,
lower-dimensionality measurements, both in terms of reconstruction accuracy and
in computational efficiency. However, AMP suffers from serious convergence
issues in contexts that do not exactly match its assumptions. We propose a new
approach to stabilizing AMP in these contexts by applying AMP updates to
individual coefficients rather than in parallel. Our results show that this
change to the AMP iteration can provide theoretically expected, but hitherto
unobtainable, performance for problems on which the standard AMP iteration
diverges. Additionally, we find that the computational costs of this swept
coefficient update scheme is not unduly burdensome, allowing it to be applied
efficiently to signals of large dimensionality.Comment: 11 pages, 3 figures, implementation available at
https://github.com/eric-tramel/SwAMP-Dem
A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
Restricted Boltzmann machines (RBMs) are energy-based neural-networks which
are commonly used as the building blocks for deep architectures neural
architectures. In this work, we derive a deterministic framework for the
training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer
(TAP) mean-field approximation of widely-connected systems with weak
interactions coming from spin-glass theory. While the TAP approach has been
extensively studied for fully-visible binary spin systems, our construction is
generalized to latent-variable models, as well as to arbitrarily distributed
real-valued spin systems with bounded support. In our numerical experiments, we
demonstrate the effective deterministic training of our proposed models and are
able to show interesting features of unsupervised learning which could not be
directly observed with sampling. Additionally, we demonstrate how to utilize
our TAP-based framework for leveraging trained RBMs as joint priors in
denoising problems
Intensity-only optical compressive imaging using a multiply scattering material and a double phase retrieval approach
In this paper, the problem of compressive imaging is addressed using natural
randomization by means of a multiply scattering medium. To utilize the medium
in this way, its corresponding transmission matrix must be estimated. To
calibrate the imager, we use a digital micromirror device (DMD) as a simple,
cheap, and high-resolution binary intensity modulator. We propose a phase
retrieval algorithm which is well adapted to intensity-only measurements on the
camera, and to the input binary intensity patterns, both to estimate the
complex transmission matrix as well as image reconstruction. We demonstrate
promising experimental results for the proposed algorithm using the MNIST
dataset of handwritten digits as example images
Inferring Sparsity: Compressed Sensing using Generalized Restricted Boltzmann Machines
In this work, we consider compressed sensing reconstruction from
measurements of -sparse structured signals which do not possess a writable
correlation model. Assuming that a generative statistical model, such as a
Boltzmann machine, can be trained in an unsupervised manner on example signals,
we demonstrate how this signal model can be used within a Bayesian framework of
signal reconstruction. By deriving a message-passing inference for general
distribution restricted Boltzmann machines, we are able to integrate these
inferred signal models into approximate message passing for compressed sensing
reconstruction. Finally, we show for the MNIST dataset that this approach can
be very effective, even for .Comment: IEEE Information Theory Workshop, 201
Efficient Per-Example Gradient Computations in Convolutional Neural Networks
Deep learning frameworks leverage GPUs to perform massively-parallel
computations over batches of many training examples efficiently. However, for
certain tasks, one may be interested in performing per-example computations,
for instance using per-example gradients to evaluate a quantity of interest
unique to each example. One notable application comes from the field of
differential privacy, where per-example gradients must be norm-bounded in order
to limit the impact of each example on the aggregated batch gradient. In this
work, we discuss how per-example gradients can be efficiently computed in
convolutional neural networks (CNNs). We compare existing strategies by
performing a few steps of differentially-private training on CNNs of varying
sizes. We also introduce a new strategy for per-example gradient calculation,
which is shown to be advantageous depending on the model architecture and how
the model is trained. This is a first step in making differentially-private
training of CNNs practical